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Abstract 

We highlight the underlying category-theoretic structure of mea- 
sures of information flow. We present an axiomatic framework in 
which communication systems are represented as morphisms, and in- 
formation flow is characterized by its behavior when communication 
systems are combined. Our framework includes a variety of discrete, 
continuous, and, conjecturally, quantum information measures. It also 
includes some familiar mathematical constructs not normally associ- 
ated with information, such as vector space dimension. We discuss 
these examples and prove basic results from the axioms. 

1 Introduction 

Information theory characterizes the transmission of information using a vari- 
ety of measures, including discrete and continuous versions of entropy, mutual 
information, and channel capacity. Quantum information theory has added 
to this list by introducing notions of quantum channel capacity. 

In this work we identify a category-theoretic structure underlying these 
measures. We show that communication systems have a natural represen- 
tation as morphisms, and using this representation, we give axioms that 
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characterize information flow. Unlike other axiomatizations of information 
[TTl El [71 dl [13], we do not require specific mathematical data such as a ran- 
dom variable or set partition. Our framework thereby encompasses not only a 
single information measure or family of measures, but a wide variety of func- 
tions used in discrete, continuous, and, conjecturally, quantum information 
theory. Vector space dimension also satisfies the axioms of an information 
measure, giving additional support to connections between information and 
dimension that have been noticed elsewhere [9l [3]. We hope our work will 
help unify and extend the results of information theory by stimulating the 
discovery of general theorems and new ways to measure information 

The organization of this paper is as follows: In section 2 we introduce the 
representation of communication systems as morphisms, and in section 3 we 
discuss how such systems may be formally combined using category-theoretic 
operations. In section 4 we present our axiomatic framework describing the 
behavior of information under these operations. Section 5 explores the var- 
ious settings to which our framework applies, and in section 6 we use the 
axioms to prove basic results about the general properties of information. 

2 Communication Systems 

We view information as something transmitted by a communication system. 
A communication system consists of a source and a destination, together with 
a method of transmitting information between them in the form of messages. 
This picture can be seen as a simplified version of Shannon's model, depicted 
in figure [1] 

We use the term "communication system" very generally. A person look- 
ing at a picture is an example of such a system, since information is trans- 
mitted from the picture to the person. Our focus here is on systems with 
mathematical representations, but the framework we present may also give 
insight into less rigorous situations. 

Communication systems have a natural representation as category-theoretic 
morphisms. They are directed relations between two objects satisfying the 
three basic properties of morphisms: a) two communication systems can be 
"composed" by passing a message through one and then the other, b) this 
composition is associative, and c) "identity" communication systems exist, 
wherein the source is the same as the destination and the message stays 
as is. We therefore model communication systems as morphisms in various 
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Figure 1: Shannon's model of a communcation system 
categories 

In most of the categories we consider here, the objects are sets with 
perhaps some additional structure, and the morphisms are set mappings pre- 
serving the structure. The elements of the sets represent possible messages, 
and the morphisms map messages sent onto messages received. Information 
can then be quantified in terms of how much the received message tells the 
receiver about which message was sent. We note, however, that our formal- 
ism may also be used with more abstract categories (see section [575]) . wherein 
the interpretation of morphisms as communication systems is less intuitive. 

3 Combining Morphisms 

We will characterize information in terms of its behavior when communica- 
tion systems are combined. However, there are several meaningful ways to 
combine communication systems, each of which corresponds to an operation 
on morphisms. One such operation is composition, as described above. In 
this section we define two new operations on morphisms and interpret them 
in terms of communication systems. 

Let C be a category. For morphisms f,g& hom(C), we define the external 
product fxg E hom (C) to be the product, if it exists, of / and g in the 
category C whose objects are morphisms in C and whose morphisms hi — »• h2 
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are commutative diagrams in C of the form 



If no such product exists, fxg is undefined. Otherwise, fxg is well-defined 
up to isomorphism in C. 

The second operation is only defined for morphisms with the same do- 
main. Let morphisms /, g have domain A. We define the internal product 
f ^aQ ^ hom (C) to be the product, if it exists, of / and g in the category Ca 
whose objects are morphisms with domain A, and whose morphisms hi /12 
are commutative diagrams in C of the form 

A 




Ci >C2. 



As above, if no such product exists, / Xa 9 is undefined; otherwise, it is 
well-defined up to isomorphism in Ca- 

To understand these operations in terms of communication systems, con- 
sider the category FinSet of finite sets and finite set mappings, where each 
morphism is interpreted as a correspondence between sent and received mes- 
sages. If / and g are set mappings with domain 5, the internal product fxsg 
is the function s 1— > (/(s), 5^(5)) evaluating / and g on the same element of S. 
This corresponds to sending the same message through two different systems 
simultaneously, fxg, on the other hand, is the map {si, S2) ^ (/(si), ^'(=^2)) 
taking two inputs and evaluating / on the first and g on the second. This 
corrpesponds to sending two different messages through different systems. 

Somewhat more generally, suppose the product of any two objects exists 
in our category C. Then the external and internal products always exist 
and are, up to isomorphism, the unique morphisms making the following 
diagrams commute: 

A< Ax B >B 



f 



X xY- 
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for external products and 



A 




X< X X Y >Y 

for internal products. 

4 Axioms for Information Functions 

We now present axioms characterizing the behavior of information measures. 
These axioms were chosen because a) they represent properties of informa- 
tion we believe to be fundamental, and b) they appear to be a minimal set 
necessary to establish basic lemmas (see section 6.) 

For a category C whose morphisms represent communication systems, we 
define an information function to be a function that quantifies the amount of 
information received by the destination as a message is sent through a system. 
Mathematically, an information function / assigns to each morphism / of C 
a nonnegative real number /(/), subject to the following axioms. 

First, if two communication systems are mathematically identical, the 
information fiow through each is the same. 

Axiom 1. Invariance: If morphisms / and g are isomorphic in C, then 

Second, if two difi^erent messages are sent through different systems, so 
that the received messages are independent of each other, the amount of 
information in both them is the sum of the information in each of them. 

Axiom 2. External Additivity: II fxg exists, I{fxg) = I{f) + I{g). 

If the same message is sent through different systems simultaneously, the 
total information gain cannot exceed the sum of the information obtained 
through each system, and may be strictly less due to redundancies in the 
received messages. More strongly, we require: 

Axiom 3. Internal Strong Suhadditivity: li f,g, and h have the same domain 
A, then 

/(/ ^ xa /i) < /(/ Xa g) + I{g Xa h) - I{g), 
if the above products exist. 
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That is, the information gained from the tripartite system f Xa 9 Xa h 
cannot exceed the amount gained from f XaO, plus that gained from gXAh, 
minus the amount I{g) that is redundant to f Xa g and g Xa h. 

The fourth axiom concerns the case where a single message is sent through 
two systems in sequence; i.e. the message received from the first system is 
relayed through the second. No new information can be gained from passing 
the message through the second system; this is known as the data processing 
inequality. Furthermore, if no information is lost in sending the message 
through the second system, then it is possible to reconstruct the first received 
message from the second. 

Axiom 4. Monotonicity: Given a diagram 

I{g ° /) < I{f) with equality if and only if there is a morphism s : C ^ B 
such that s o g o f — f. 

Finally, the amount of information that can be sent through a system is 
limited the range of messages which can be received. One cannot improve on 
a system in which the sent messages match the received messages perfectly. 

Axiom 5. Destination Matching: For any morphism / with codomain B, 
I{f) < /(ids). 

5 Examples of Information Functions 

We now explore mathematical information functions encompassed by our 
framework, including many classical information measures as well as familiar 
constructs not usually seen as related to information. We also show how 
our framwork might be applied to quantum channel capacities. With the 
exception of these quantum capacities, satisfaction of the axioms is either a 
well-known result or a simple exercise. 

5.1 Discrete Communication 

In the simplest mathematical models of communication, there is a finite set 
of possible messages to send and a deterministic mapping from messages sent 
to messages received. Such systems can be represented by morphisms in the 
category FinSet of finite sets and set mappings. 
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5.1.1 Hartley Entropy 



The information flow through a discrete system can be quantified in terms of 
the number of possible received messages. The greater variety in the mess- 
sages which can be received, the more information is gained upon receiving a 
particular message. One simple way to measure this variety is the following: 
for a set mapping f : A —>■ B, we define the Hartley entropy of / to be 



the logarithm (customarily taken base 2) of the cardinality of /'s image. 
Hartley entropy measures the expected number of bits needed to encode a 
received message if all possible received messages are equally likely. 

5.1.2 Shannon Entropy 

The Shannon entropy modifies Hq to incorporate unequal probabilities of 
received messages. For a mapping f : A B of finite sets, we define the 
Shannon entropy of / to be 



If all sent messages a are equally likely, H{f) quantifies the uncertainty of 
the received message as the expected number of bits needed to encode /(a) 
using an optimal encoding. 

By a theorem of Aczel, Forte, and Ng [2], the only information functions 
on the category FinSet are linear combinations of the Shannon and Hartley 
entropies. 

5.1.3 Noisy Information 

The category FinSet has limitations as a setting for modelling discrete com- 
munication systems. There is no way to represent communication errors 
induced by noise, or to record the fact that some messages are more likely to 
be sent than others. We can overcome both of these limitations by consid- 
ering our objects to have two parts: a visible part representing the messages 
that can be sent or received, and a hidden part representing noise that may 
affect transmission. 



Ho{f)=log\f{A)\, 




\f-Kb)\ 
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Let NoisyFinSet be the category whose objects are pairs (M, A) of finite 
sets, together with a surjective set map tt^ : M A. For each a E A, the 
preimage Ma = 7r^^(a) C M represents the environmental noise factors that 
might be sent along with a. 

Morphisms in this category are diagrams 

M— ^AT 
A B. 

For each intended message a E A, the actual transmitted data is an element 
of the preimage Ma, representing both the original message and the noise. 
Different elements in this preimage may map to different elements of B under 
ttb o f, in accordance with the possibility that noise may change the received 
message. The map ttb may be used to model the decoding of an error- 
correcting code. 

For such a morphism we define the noisy Shannon information in the fol- 
lowing manner: Consider M as a probability space with normalized counting 
measure (that is, consider each element of M as equally likely to occur.) 
Then tta and ttb o / can be interpreted as random variables with values in A 
and B respectively. Define NI{f) = /(tt^ o /;7ryi), the mutual information 
of these two variables. This quantity may be computed precisely as 

^^(/) = E n ^1]^ - 21^1 log 

where is the preimage f^^{Ni,) = f^^iT^B^ib)) C M, i.e. the set of all 
noise elements m E M that cause message b to be received. This function 
generalizes the Shannon entropy as defined above. If an element m e M is 
chosen randomly with uniform probability, then NI{f) equals the average 
amount of Shannon information that the received message tt^ o /(m) G B 
imparts about the sent message 7r^(m). This formalism also allows for un- 
equal probability in sent messages: the probability of a message a E A being 
sent is proportional to the cardinality of the preimage M^. 

5.1.4 Channel Capacity 

Channel capacity is the maximum amount of information that can be sent 
through a noisy system, where the maximum is taken over all probability 
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distributions on the set of sent messages. To represent channel capacity as 
an information function, we again use the categrory NoisyFinSet. Consider 
a morphism 

/ 



M- 



A B. 

For any message a e A we consider each element of the preimage 7r^^(a) to 
be equally likely. Using this rule, given any probability distribution pA on 
A, we can associate a probability distribution on M, and, via ns o f, joint 
distribution p on Ax B and a marginal distribution ps on B. We define the 
channel capacity C{f) to be the maximum over all probability distributions 
on A of the mutual information between the sent message a & A and the 
received message beB: 



= niax 5^ p{a, h) log ^^^y^ 



beB 



PA{a)pB{b) ' 



5.2 Continuous Communication 

Many real-world communication systems have a continuous range of mes- 
sages that can be sent or received. To model these systems we use the 
category Prob whose objects are probability spaces, i.e. measure spaces 
(M, /x) with //(M) = 1. We define the morphisms in Prob to be measurable 
functions / : (M, fx) — > {N, i/) that are backwards measure preserving, i.e. 
IJ,{f~^{U)) = iy{U) for each measurable U G N. This condition guarantees 
that the probability measure induced by / on iV agrees with ly. 

Internal products of morphisms are not guaranteed to exist in this cat- 
egory. The obvious candidate, x i— > {f{x),g{x)), does not in general satisfy 
the backwards measure preserving condition. External products, however, do 
exist: for / : (Mi,/zi) — > (Af2,/U2) and g : {Ni,ui) — > (A^2,'^2), the external 
product fxg is the natural map between the product spaces Mi x A^^i and 
M2 X N2, sending {x,y) to {f{x),g{y)). It is easily verified that this map is 
backwards measure preserving. 
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5.2.1 Noisy Information 



To represent imperfect communication, we add "noise" to this category in 
the same manner as for FinSet, by forming a category NoisyProb whose 
objects are pairs ((M, (A, a)) with morphisms tt^ : M — >■ A. Morphisms 
in NoisyProb are again given by commutative diagrams 

(M,/i)^(iV,z.) 



Such a morphism / induces a surjective measurable map f : M ^ Ax B 
sending m e M to (7r^(m), tt^ o f{m)). f and the measure on M induce a 
probability measure p on A x B. If p is absolutely continuous with respect 
to fj, X u, we define the (continuous) noisy Shannon information of / to be 



NIcont{f)= I In— ^ dp, 
Jaxb d{p X v) 



where ^^^^^-^ is the Radon-Nikodym derivative of p with respect to pxv. If p is 
not absolutely continuous with respect io pxv, we leave /(/) undefined. Our 
definition agrees with the usual formula for continuous mutual information 



/ P[x, y) In — ' d[p X v) 
Jaxb PA{x)PB{y) 



under the substitutions p{x, y) = j^jj^ and Pa{x) = psiv) = 1 with respect 
to the probability measures a and p, respectively. 

(It may come as a surprise that NIcont for a noiseless system, wherein 
tta is an isomorphism, is usually undefined if A contains uncountably many 
points. This is because a communication system that can faithfully transmit 
any of uncountably many inputs can send any finite amount of information 
in a single message.) 



5.2.2 Channel Capacity 

Channel capacity is obtained by maximizing the amount of transmitted in- 
formation over the set of all probability distributions on the space of sent 
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messages. We represent such probability distributions by measurable func- 
tions p : A ^ ]R>o with J^p da = 1. For a particular message a & A, we 
view all environmental noise factors in the preimage n^^(a) as equally likley; 
we therefore define a new probability measure Hp on M by 

l^piU) = / pOTTAd/I 

Ju 

for each measurable U C M. 

Given a morphism / as above, the map f : M ^ A x B induces a 
probability measure pp on A x B. The continuous channel capacity of / is 
defined as 

Ccontif) = sup / In /^^^ dpp, 
P Jaxb d[p. xu) 

if such a supremum exists; otherwise C(/) is undefined. 



5.3 Quantum Communication 

For systems that communicate using processes governed by quantum me- 
chanics, the transmission of information is described by quantum informa- 
tion theory. Quantum information theory is a young field with many open 
problems; in particular, we cannot say at this time whether the various ca- 
pacities of a quantum channel satisfy our axioms for information functions. 
However, our category-theoretic framework suggests new problems and casts 
existing ones in a new light. 

Consider a finite-dimensional quantum system represented by Hilbert 
space H. A "message" in this system is represented by a density matrix: 
a positive-semidefinite Hermitian operator of trace one acting on H. Let 
D{H) denote the space of all density matrices over the Hilbert space H. 
Given two such spaces D{Hi) and D{H2), the natural choice for a morphism 
between them is a completely positive trace-preserving linear map. We call 
such maps CP m,aps or quantum channels. We denote by Quant the category 
whose objects are spaces D[H) of density matrices over finite-dimensional 
Hilbert spaces, and whose morphisms are GP maps. 

External products in this category are given by tensor product of GP 
maps. Internal products do not exist in all cases; for instance, the internal 
product of the identity map p ^ p with itself would have to he p \—> p (E) p, 
but this is nonlinear and violates the no-cloning theorem. However, there 
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are important cases where internal products do exist; for example, given a 
bipartite system AB, the internal product of the partial traces Tr^ : D{A<S) 
B) D{A) and Tr^ : D{A ^ B) ^ D{A) is the identity map D{A®B)^ 
D{A(g)B). 

There are several different notions of capacity for a quantum channel, 
depending on the intended application. The quantum capacity measures the 
capacity of the channel to send quantum states intact. The classical capac- 
ity measures the capacity to send classical information. The entanglement- 
assisted classical capacity is the capacity to send classical information if the 
sender and receiver are allowed to share an arbitrary number of entangled 
quantum states prior to transmission. For mathematical definitions of these 
quantities we refer our readers to the literature [IHl S] • 

All the above capacities satisfy invariance, monotonicity, and destination 
matching. External additivity for these capacities is a famous open prob- 
lem; it amounts to the question of whether quantum entangled messages can 
be communicated more efficiently than unentangled messages. For partial 
results on this problem, see 0, [12]. Internal strong subadditivity of these 
capacities has not, to our knowledge, been studied. This question is likely 
related to the strong subadditivity of Von Neumann entropy, proven by Lieb 
and Ruskai [8]. 

5.4 Vector Communication 

There are strong reasons to suspect a relationship between information and 
dimension. For example, the dimension of a vector space equals the number 
of coordinates required to specify a point in the space, much like the Shannon 
entropy equals the average number of bits needed to specify the value of a 
random variable. Deep connections have also been found between notions of 
dimension and complexity [9l [3]. It is therefore not surprising that vector 
space dimension is also an information function. 

Suppose we have a communication system in which the set of possible 
messages is represented by a finite-dimensional vector space, and sent mes- 
sages are mapped linearly onto received messages. We represent such systems 
using the category /^-FinVect of finite-dimensional vector spaces over a field 
K, with morphisms given by linear maps. For a morphism f : V ^ W, we 
define 

d{f)= dim f{V). 
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d{f) can be interpreted as the amount by which knowledge of f{v) e W 
reduces the dimensional uncertainty of v & V. 



5.5 Information Functions on Dual Categories 

Finally, we present a pair of information functions whose interpretation in 
terms of communication systems is not clear. The categories for these are the 
duals of the familiar categories FinSet and /C-FinVect. In a dual category, 
the internal and external product operations correspond to similarly defined 
internal and external coproduct operations in the original category. 

5.5.1 Cardinality of Image 

Consider the category FinSet* whose objects are finite sets and whose mor- 
phisms f* : A ^ B correspond to set mappings f : B A. The external 
product of f* : A ^ B and g* : C ^ D is the morphism f*xg* : AUC ^ 
B L\ D {U denotes disjoint union) corresponding to the set mapping 



The internal product of f* : A ^ B and g* : A ^ C is the morphism 
f*xg* : A ^ B L\ C corresponding to the set mapping 



The function mapping /* to \f{B)\, the cardinality of the image of the 
corresponding set map /, is an information function in this category. It 
is unclear, however, in what sense the dual of a set mapping represents a 
communication system. 

5.5.2 Dimension of Image 

Similarly, we can explore the category i^-FinVect* whose objects are finite 
dimensional vector spaces and whose morphisms f* : V ^ W correspond 



f+g 



BUD 



AUC 




f+AQ-BUC^A 
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to linear maps f : W V. The external product of f* : V ^ W and 
g* : T ^ U is the morphism f*xg* : V ®T ^ W ® U corresponding to the 
linear map 

f+g:W®U ^V®T 

f+g : w © I— > f{w) © g{u) ior w eW,u E U. 

The internal product oi oi f* : V ^ W and g* : V ^ U is the morphism 
f*xg* : V ^ W ® U corresponding to the linear map 

f+vg-.wou^v 

f +v g : w ®ui-^ f{w) + g{u) ior w eW,u E U. 

For a morphism f* : V ^ W, the function /(/*) = dim(/(W^)) is an 
information function in this category. Since dimension of image is also a 
information function in the original cageory i^'-FinVect, this function might 
be called a hi- information function^ in that it is an information function on 
a category and its dual. The existence of a bi-information function for K- 
FinVect is doubtless related to the existence of biproducts in this category. 

6 Basic Results 

We now establish some basic facts about information functions. We start 
by showing that products and compositions are well-defined on isomorphism 
classes in C and Ca- 

Proposition 1. Suppose a ^ b and c'^ d in C, and e = f and g ^ h in Ca- 
Then, assuming the following compositions and products exist, 

(a) aoc^bodinC. 

(b) axc = bxd in C. 

(c) aoe^bofin Ca- 

(d) eXAg^ f XaH inCA- 

Proof. Exercise. □ 
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Now let / be an arbitrary information function. The following proposition 
says that no communication system can do better than one which reproduces 
the inputs exactly. 

Proposition 2. Source matching: For any morphism f with domain A, 
/(/)</(id^). 

Proof. /(/) = /(/ o id^) < /(idyl) by monotonicity. □ 

Proposition 3. Monotonicity with respect to internal products: Let f and 
g be morphisms with domain A. Assuming the relevant products exist, 

(a) I{fxAg)>I{f) 

(b) /(/x^ /)=/(/). 

In other words, you will never lose any information by sending the same 
message simultaneously through two different systems versus sending it through 
just one of them, but if the two systems are identical, you won't gain any 
information either. 

Proof, (a) By the definition of products there is a morphism tti : f XaQ ^ 
/ in C^, which corresponds to a diagram 

A 

fxAg V 
C >B. 

The result now follows from monotonicity. 

(b) We must now show /(/ Xa f) < I if)- Again by the definition of 
products, there is a unique morphism S : f ^ f x f in Ca making the 
following diagram commute: 
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The morphism 5 represents a diagram 



A 



f 




fXAf 



B 



from which the result follows by monotonicity. 



□ 



Further results depend on the existence of terminal objects in our cate- 
gory C. A object T is terminal if there is exactly one morphism into T from 
any object A. Any singleton set is a terminal object in FinSet. Intuitively, 

a destination represented by a terminal object is unable to discriminate be- 
tween messages, and thus cannot receive information. We prove this and 
other results in the following propositions: 

Proposition 4. Let T be a terminal element of C. For a fixed object A, let t 
be the unique morphism t : A T. Then, assuming the appropriate products 

exist, 

(a) idr and t are terminal elements in C and Ca, respectively. 

(b) For any morphism g e homC, gx idy g in C. If g has domain A, 
then g Xa t ^ g in Ca- 

(c) Let B be an object for which Ax B exists. Then ttb — tx ids in Caxb- 
Proof, (a) Exercise. 

(b) Combine part (a) with the fact that A x T = A for any object A, in 
any category with terminal object T. 

(c) Let s and t be the unique morphisms t : A ^ T, s : B ^ T. Let e 
be the unique morphism e : B ^ T x B making the following diagram 
commute: 



B 




T^ 



Tx B 



^B 
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Then the diagrams 

Ax B 




T< T X B >B 

and 

AxB 




T< T X B >B 

also commute. By the definition of products we must have e o tt^ = 
tx ids, and the result now follows from the fact that e is an isomor- 
phism. □ 

Corollary 5. Let f : A C be any morphism, and let e and t be as above. 
Then 

(a) /(/x idr) = I{f t) = I{f) for any morphism f , whenever these 
products are defined. 

(b) /(idT)=/(t)=0. 

(c) Subadditivity: /(/ x^ g) < /(/) + 1(g) for any two morphisms f,g 
with domain A. 

(d) // the product A x B exists for an object B, then I{f o tt^) = I{f)- 

Part (d) says that the information fiow through a communication system 
/ is not affected by the presence of "irrelevant" information in B. 

Proof. Parts (a) follows direcly from the proposition. For part (b), /(idr) = 
follows from part (a) and additivity, then I{t) = by destination matching. 
Part (c) is proven by subsituting t into strong subadditivity, 

/(/ xa txAg) + lit) < I{f xa t) + I{t xa g), 

and invoking previous results. For part (d), /(tt^) = /(id^) by Proposition 
m^c), and the result follows from Proposition [UJ^a). □ 
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7 Conclusion 



Mathematical abstractions like groups and topological spaces have the power 
to illuminate connections between different objects of study and inspire the 
discovery of new objects. We hope this present abstraction will lead to the 
discovery of new ways to measure information, and deepen understanding of 
information's general properties. In the future, we aim to prove deeper results 
and investigate information functions relevant to specific applied situations. 
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